Repository navigation
fix(storybook-desktop): report all-skipped WDIO runs as incomplete instead of passed - #4369
udit (UditDewan) wants to merge 2 commits into
Conversation
A WDIO selection in which every case called skip(...) was written to run.json as "passed" and exited successfully, so it could satisfy a required test gate without exercising the product. runWdioStoryTests now records such runs as "incomplete" and throws a WdioStoryCoverageUnavailableError (distinct from WdioStoryTestRunError) that carries every result and skip reason. Smoke runs treat a single all-skipped story as non-fatal, but report the aggregate as incomplete and fail when no test across the run passed. Both reports now include selected/passed/skipped/failed counts. Mixed pass/skip runs still pass. Fixes microsoft#4342
Bundle size reportTree-shaken, minified production esbuild bundles with React and React Native runtimes externalized.
The job is advisory: size changes are reported but do not fail the pull request. Bundle or analysis errors still fail. |
| export function countWdioStoryTests(results: readonly { status: string }[]): WdioStoryTestCounts { | ||
| const passed = results.filter(({ status }) => status === 'passed').length; | ||
| const skipped = results.filter(({ status }) => status === 'skipped').length; | ||
| return { selected: results.length, passed, skipped, failed: results.length - passed - skipped }; |
There was a problem hiding this comment.
[P2] Preserve the selected count when infrastructure aborts the remaining cases
results.length counts recorded outcomes, not the selected cases. runWdioStoryTests computes all selected cases before execution, but its outer catch stops the loop on a connection or session-cleanup error. With two selected callbacks and a connection failure on the first, the new report says selected: 1, failed: 1, omitting the second selected callback entirely. Grouped smoke likewise breaks before later selected stories run. This loses the selected-versus-executed distinction needed to interpret missing coverage (and requested in #4342). Could the callers supply the actual selection size separately, retaining the outcome counts and making unrun cases distinguishable? Please add a regression with multiple selected cases and an infrastructure failure before the remaining cases execute.
Platforms Impacted
Description of changes
Fixes #4342.
A WDIO selection in which every case called
skip(...)was written torun.jsonas"status": "passed"and the command exited successfully, so an all-skipped run (for examplecomponents-checkbox--defaultwithout physical pointer input) could satisfy a required test gate without exercising the product.src/testing/runWdioTests.tsWdioStoryCoverageUnavailableError, thrown when no selected case passed and none failed. It is deliberately not aWdioStoryTestRunError, so coverage unavailability stays distinguishable from an assertion failure. It carries every result, and its message lists the skip reasons.run.jsonstatus is nowfailed/incomplete/passed, and the report includescounts: { selected, passed, skipped, failed }. Per-testskipReasons are unchanged.yarn storybook testtherefore exits non-zero for an all-skipped selection.src/cli/smokeTests.tsincompleteand the run fails. The smokerun.jsonalso gainscounts.Existing zero-selection, failure, timeout, crash, evidence, and session-cleanup paths are untouched; the new checks only run after them and only when there are no failures.
Out of scope here: CI summary rendering and required-cohort promotion policy (#4349/#4345). They can key off the new
incompletestatus andcounts.Verification
yarn build,yarn lint,yarn testin@fluentui-react-native/storybook-desktop(163 tests);yarn format:check;yarn lage test-links.status: 'incomplete'with counts and skip reason{ selected: 2, passed: 1, skipped: 1 }fixture-button--skipnow rejects withWdioStoryCoverageUnavailableErrorandrun.jsonisincomplete. The existing pass, fail/timeout/crash, named, mixedrestrictedButton, and empty-selection contract cases are unchanged.mainand pass with this change.run.json"status": "passed", exit 0"status": "incomplete"with counts, non-zero exit, skip reasons in the errorPull request checklist
This PR has considered (when applicable):